Image processing device, method and program

ABSTRACT

A three-dimensional moving image and an ultrasonic moving image showing a body part making periodic motion are obtained, and, from the moving images, a characteristic part having a shape that changes with the periodic motion is extracted. Phases of the periodic motion captured in the moving images are obtained. For at least one of the moving images, the phases are obtained based on the shape of the extracted characteristic part. For each phase, the positions of the characteristic part shown in the three-dimensional moving image and the ultrasonic moving image are associated with each other based on the extracted characteristic part and the obtained phases. A superimposed image is generated by aligning, for each phase, the positions of the characteristic part shown in the three-dimensional moving image and the ultrasonic moving image with each other based on the associated positions of the characteristic part and the phases, and displayed.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing device, an imageprocessing method and an image processing program for aligning anddisplaying a three-dimensional moving image taken with a CT or MRapparatus with a two-dimensional or three-dimensional moving image takenwith an ultrasonic diagnostics apparatus.

2. Description of the Related Art

In the medical field, detailed three-dimensional moving images, whichare obtained by imaging a body part of a patient making a predeterminedperiodic motion, such as the heart, with a modality having high spatialresolution and contrast resolution, such as a CT or MR apparatus, arewidely used for imaging diagnosis.

On the other hand, ultrasonic moving images taken with an ultrasonicdiagnostics apparatus, which has no problem of radiation exposure, etc.,allows examination with a simple device, and provides information aboutblood flow based on reflection of ultrasonic waves by the blood flow,are also effective for imaging diagnosis. Further, along with thedevelopment of the ultrasound diagnostic technique, three-dimensionalultrasonic moving images of subjects have become available as theultrasonic moving images, in addition to conventional ultrasonic movingimages obtained with respect to a predetermined two-dimensional crosssection of subjects.

In order to use advantages of both the above-described moving images,doctors conduct the imaging diagnosis with displaying both the movingimages in a state where they show the same phase of heart beat based onelectrocardiographic data that is obtained during imaging of each movingimage. At this time, the user (doctor) references one of the images withmanually changing the position and direction shown in the other of theimages to be the same as those shown in the one of the images so thatthe moving images showing the same position and phase are displayed atthe same time on a display in a manner allowing comparison therebetween.However, while a CT or MR image is taken in fixed position andorientation relative to the subject, an ultrasonic image is taken withpressing an ultrasound probe against the subject at an arbitrary angle.Therefore, it is difficult to identify the position and direction of theultrasonic image relative to the subject. Further, it is necessary toidentify a frame image forming the CT or MRI moving image and a frameimage forming the ultrasonic moving image that correspond to the samephase. Therefore, it imposes significant time and labor on the user todisplay these moving images in a manner allowing comparisontherebetween.

Japanese Unexamined Patent Publication No. 2003-153877 (hereinafter,Patent Document 1) has proposed a technique which involves: extracting apredetermined characteristic part, such as a blood vessel area includinga blood flow image, from ultrasonic image data of an examined body part;aligning the position of the predetermined characteristic part shown inthe ultrasonic image with the position of the predeterminedcharacteristic part shown in an MR image obtained in advance; correctingthe MR image such that the MR image and the ultrasonic image show thepredetermined characteristic part in the same position; andsuperimposing the corrected MR image and the ultrasonic image anddisplaying the superimposed image on a display device.

Further, Japanese Unexamined Patent Publication No. 2009-022459(hereinafter, Patent Document 2) has proposed a technique whichinvolves: synchronizing timing of a three-dimensional CT moving imagewith timing of a three-dimensional ultrasonic moving image based onelectrocardiographic data; reconstructing the ultrasonic image bytransforming a spatial coordinate system of the ultrasonic moving imageinto a spatial coordinate system of the CT moving image using atransformation matrix; and displaying the reconstructed moving imagesbeing aligned and superimposed.

According to the technique taught in Patent Document 1, it is able tospatially align a three-dimensional still image taken with a CTapparatus with an ultrasonic still image. However, with the techniquetaught in Patent Document 1, a three-dimensional moving image taken witha CT apparatus and an ultrasonic moving image cannot be associated withthe phases of heart beat, and therefore it is difficult to display thesuperimposed moving images such that they show the same phase and thesame spatial position. Further, according to the technique taught inPatent Document 2, it is necessary to obtain electrocardiographic datacorresponding to the moving images to align the moving images with eachother with respect to the phase of heart beat. Therefore, in a casewhere the electrocardiographic data corresponding to one of or both ofthe moving images is not available, it is difficult to associate thecorresponding phases of heart beat shown in the moving images with eachother.

SUMMARY OF THE INVENTION

In view of the above-described circumstances, the present invention isdirected to providing an image processing device, an image processingmethod and an image processing program that facilitate aligning athree-dimensional moving image taken with a CT or MR apparatus with anultrasonic moving image with respect to phases of a periodic motion andthe position thereof to generate and display a superimposed image of thethree-dimensional moving image taken with a CT or MR apparatus and theultrasonic moving image.

An aspect of the image processing device according to the invention isan image processing device including: image obtaining means forobtaining a three-dimensional moving image showing a body part of apatient and an ultrasonic moving image showing the body part, the bodypart making a predetermined periodic motion; characteristic partextracting means for extracting, from a plurality of frame imagesforming the obtained three-dimensional moving image and ultrasonicmoving image, a predetermined characteristic part having a shape thatchanges in response to the periodic motion; phase obtaining means forobtaining phases of the periodic motion captured in thethree-dimensional moving image and the ultrasonic moving image, wherein,for at least one of the three-dimensional moving image and theultrasonic moving image, the phases are obtained based on the shape ofthe extracted characteristic part; associating means for associating,for each phase, a position of the characteristic part in a frame imageforming the three-dimensional moving image corresponding to the phasewith a position of the characteristic part in a frame image forming theultrasonic moving image corresponding to the phase based on theextracted characteristic part and the obtained phases; image generatingmeans for generating a superimposed image of the three-dimensionalmoving image and the ultrasonic moving image by aligning, for eachphase, a position of the characteristic part in a frame image formingthe three-dimensional moving image corresponding to the phase with aposition of the characteristic part in a frame image forming theultrasonic moving image corresponding to the phase based on theassociated positions of the characteristic part and the phases; anddisplay controlling means for displaying the generated superimposedimage on a display device.

An aspect of the image processing method according to the invention isan image processing method including: obtaining a three-dimensionalmoving image showing a body part of a patient and an ultrasonic movingimage showing the body part, the body part making a predeterminedperiodic motion; extracting, from a plurality of frame images formingthe obtained three-dimensional moving image and ultrasonic moving image,a predetermined characteristic part having a shape that changes inresponse to the periodic motion; obtaining phases of the periodic motioncaptured in the three-dimensional moving image and the ultrasonic movingimage, wherein, for at least one of the three-dimensional moving imageand the ultrasonic moving image, the phases are obtained based on theshape of the extracted characteristic part; associating, for each phase,a position of the characteristic part in a frame image forming thethree-dimensional moving image corresponding to the phase with aposition of the characteristic part in a frame image forming theultrasonic moving image corresponding to the phase based on theextracted characteristic part and the obtained phases; generating asuperimposed image of the three-dimensional moving image and theultrasonic moving image by aligning, for each phase, a position of thecharacteristic part in a frame image forming the three-dimensionalmoving image corresponding to the phase with a position of thecharacteristic part in a frame image forming the ultrasonic moving imagecorresponding to the phase based on the associated positions of thecharacteristic part and the phases; and displaying the generatedsuperimposed image on a display device.

An aspect of the image processing program according to the invention isan image processing program for causing a computer to function as: imageobtaining means for obtaining a three-dimensional moving image showing abody part of a patient and an ultrasonic moving image showing the bodypart, the body part making a predetermined periodic motion;characteristic part extracting means for extracting, from a plurality offrame images forming the obtained three-dimensional moving image andultrasonic moving image, a predetermined characteristic part having ashape that changes in response to the periodic motion; phase obtainingmeans for obtaining phases of the periodic motion captured in thethree-dimensional moving image and the ultrasonic moving image, wherein,for at least one of the three-dimensional moving image and theultrasonic moving image, the phases are obtained based on the shape ofthe extracted characteristic part; associating means for associating,for each phase, a position of the characteristic part in a frame imageforming the three-dimensional moving image corresponding to the phasewith a position of the characteristic part in a frame image forming theultrasonic moving image corresponding to the phase based on theextracted characteristic part and the obtained phases; image generatingmeans for generating a superimposed image of the three-dimensionalmoving image and the ultrasonic moving image by aligning, for eachphase, a position of the characteristic part in a frame image formingthe three-dimensional moving image corresponding to the phase with aposition of the characteristic part in a frame image forming theultrasonic moving image corresponding to the phase based on theassociated positions of the characteristic part and the phases; anddisplay controlling means for displaying the generated superimposedimage on a display device.

The “body part” herein may be any body part that makes a predeterminedperiodic motion, and a typical example thereof is the heart. The“predetermined periodic motion” herein may be any repeatable motionwhere each part included in the body part moves in a predetermineddirection within a predetermined range, such as heart beat, respirationin lungs, flexion and extension of a joint, etc. Further, in theinvention, in the case where the body part making the periodic motion isthe heart, the characteristic part may be any of the ventricles, theatriums, the muscles, the valves and the apex of the heart. In thiscase, the phase obtaining means may obtain the phases based on a stateof opening and closing of any of the valves of the heart. The phaseobtaining means may optionally obtain the phases by identifying the endof diastole and/or systole of the heart based on the state of openingand closing of the mitral valve and/or the aortic valve among the valvesof the heart.

In the invention, the three-dimensional moving image may be anythree-dimensional moving image that shows the shape of a body part of asubject, and an example thereof is a three-dimensional moving imagetaken with a CT or MRI apparatus.

In the invention, the ultrasonic moving image may be a three-dimensionalmoving image or a moving image showing a cross section including thecharacteristic part.

The “phases of the periodic motion” herein refers to stages of theperiodic motion, and the number of phases for one period of the periodicmotion may be any number.

In the invention, the characteristic part extracting means mayautomatically extract the characteristic part.

In the invention, for at least one of the three-dimensional moving imageand the ultrasonic moving image, the phase obtaining means may obtainthe phases from accompanying information of the moving image.

In the invention, the image generating means may generate thesuperimposed image by obtaining pixel spacing from accompanyinginformation of each of the three-dimensional moving image and theultrasonic moving image and providing the moving images with the samepixel spacing based on the obtained pixel spacing.

In the invention, the image generating means may generate thesuperimposed image by superimposing, on the three-dimensional movingimage, the ultrasonic moving image shown by a color Doppler method basedon Doppler shift of blood flow.

According to the image processing device, method and program of theinvention, a three-dimensional moving image showing a body part, whichmakes a predetermined periodic motion, of a patient and an ultrasonicmoving image showing the body part are obtained; from a plurality offrame images forming the obtained three-dimensional moving image andultrasonic moving image, a predetermined characteristic part having ashape that changes in response to the periodic motion is extracted;phases of the periodic motion captured in each of the three-dimensionalmoving image and the ultrasonic moving image are obtained, wherein, forat least one of the three-dimensional moving image and the ultrasonicmoving image, the phases are obtained based on the shape of theextracted characteristic part; for each phase, a position of thecharacteristic part in a frame image forming the three-dimensionalmoving image corresponding to the phase is associated with a position ofthe characteristic part in a frame image forming the ultrasonic movingimage corresponding to the phase based on the extracted characteristicpart and the obtained phases; a superimposed image of thethree-dimensional moving image and the ultrasonic moving image isgenerated by aligning, for each phase, a position of the characteristicpart in a frame image forming the three-dimensional moving imagecorresponding to the phase with a position of the characteristic part ina frame image forming the ultrasonic moving image corresponding to thephase based on the associated positions of the characteristic part andthe phases; and the generated superimposed image is displayed on adisplay device. Therefore, even when there is no electrocardiographicdata available, the superimposed image of the three-dimensional movingimage and the ultrasonic moving image can easily be generated based onthe shape of the characteristic part. With this superimposed image, theuser can understand the object of observation by compensating for lowresolution areas of the ultrasonic moving image with the high spatialresolution of the three-dimensional moving image, and can easilyunderstand the information that is obtained only from the ultrasonicmoving image at the same time. Therefore, the user can efficiently andaccurately conduct the imaging diagnosis.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating the schematic configuration of an imageprocessing device according to one embodiment of the present invention,

FIG. 2 is a diagram illustrating the flow of a process carried out bythe image processing device according to one embodiment of theinvention,

FIG. 3 is a diagram illustrating an example of a superimposed imagedisplayed by the image processing device according to one embodiment ofthe invention, and

FIG. 4 is a diagram illustrating an example of a superimposed imagedisplayed by a modification of the image processing device according toone embodiment of the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of an image processing device, an imageprocessing program and an image processing method of the presentinvention will be described in detail with reference to the drawings.

FIG. 1 illustrates the schematic configuration of a hospital system 1incorporating an image processing device 6 according to one embodimentof the invention. The hospital system 1 includes an examination roomsystem 3, a data server 4 and a diagnosis workstation (WS) 6, which areconnected with each other via a local area network (LAN) 2.

The examination room system 3 includes various modalities 32 for imaginga subject, and an examination room workstation (WS) 31 used for checkingand controlling images outputted from the individual modalities. Themodalities 32 in this example includes a CT (Computed Tomography)apparatus and an MRI (Magnetic Resonance Imaging) apparatus, which areable to obtain a shape image representing shape information of theheart, and also includes an ultrasonic diagnostics apparatus, etc. Amongthese modalities 32, the CT apparatus and the MRI apparatus arecompliant to the DICOM (Digital Imaging and Communication in Medicine)standard, and output the obtained volume data as a DICOM file withadding accompanying information.

The file outputted from each modality 32 is transferred to the dataserver 4 by the examination room WS 31. The data server 4 is formed by acomputer with relatively high processing capacity including ahigh-performance processor and a mass memory, on which a softwareprogram for providing the function of a database management system(DBMS) is implemented. The program is stored in a storage, loaded in thememory upon startup, and executed by the processor. The data server 4causes the file transferred from the examination room WS 31 to be storedin a mass storage 5. Further, in response to a request to search fromthe diagnosis WS 6, the data server 4 selects a file that meets a searchcondition from files stored in the mass storage 5 and sends the file tothe diagnosis WS 6.

The diagnosis WS 6 is formed by a general-purpose workstation includinga standard type processor, a memory and a storage, on which the imageprocessing program for assisting diagnosis is implemented. The imageprocessing program is installed on the diagnosis WS 6 from a recordingmedium, such as a DVD, or downloaded from a server computer connectedvia the network before being installed. A display 7 and an input device8, such as a mouse and a keyboard, are connected to the diagnosis WS 6.

The image processing program implemented on the diagnosis WS 6 is formedby sets of program modules for accomplishing various functions. Amongthem is a set of program modules for accomplishing the image processingfunction. The program is stored in the storage, loaded in the memoryupon startup, and executed by the processor. With this, the diagnosis WS6 operates as: image obtaining means 61 for obtaining athree-dimensional moving image V1 showing a body part (the heart), whichmakes a predetermined periodic motion, of a patient and an ultrasonicmoving image V2 showing the body part; characteristic part extractingmeans 62 for extracting, from a plurality of frame images forming theobtained three-dimensional moving image V1 and ultrasonic moving imageV2, a predetermined characteristic part (mitral valve MV) having a shapethat changes in response to the periodic motion; phase obtaining means63 for obtaining phases of the periodic motion captured in thethree-dimensional moving image V1 and the ultrasonic moving image V2,wherein, for at least one of the three-dimensional moving image V1 andthe ultrasonic moving image V2, the phases are obtained based on theshape of the extracted characteristic part; associating means forassociating, for each phase, the position of the characteristic part ina frame image forming the three-dimensional moving image correspondingto the phase with the position of the characteristic part in a frameimage forming the ultrasonic moving image corresponding to the phasebased on the extracted characteristic part and the obtained phases;image generating means 65 for generating a superimposed image of thethree-dimensional moving image and the ultrasonic moving image byaligning, for each phase, a position of the characteristic part in aframe image forming the three-dimensional moving image V1 correspondingto the phase with a position of the characteristic part in a frame imageforming the ultrasonic moving image V2 corresponding to the phase basedon the associated positions of the characteristic part and the phases;and display controlling means 66 for displaying the generatedsuperimposed image on the display device 7.

FIG. 2 is a flow chart illustrating the flow of image processing of thisembodiment. FIG. 3 shows an example of the displayed superimposed image.Now, the flow of a process carried out by the functions of the WS 6(image processing device) of this embodiment is described in detailusing FIGS. 2 and 3. This embodiment is described in conjunction withthe case of heart examination as an example.

Prior to the process of this embodiment, during the heart examination, amoving image of the chest of the subject including one period of heartbeat is taken using a CT apparatus, or the like, and the thus takenthree-dimensional moving image V1 (volume data) with the accompanyinginformation added thereto is transferred as a DICOM file to the dataserver 4 and stored in the mass storage 5. The volume data is formed bya collection of pieces of voxel data representing a density distributionin a three-dimensional space. In each voxel data, X-ray absorption, orthe like, is indicated as a voxel value. Further, a moving image of thechest of the same subject is taken by transesophageal echocardiography(TEE), which is ultrasound imaging carried out by inserting anultrasound probe through the mouth to the esophagus, and the thus takenthree-dimensional ultrasonic moving image V2 is transferred to the dataserver 4 and stored in the mass storage 5.

First, when an image processing function for the heart is selected on aninitial screen and the patient ID number, the examination number, etc.,are inputted on a predetermined input screen, the image obtaining means61 sends the inputted information to the data server 4 and sends arequest to search for and transfer the corresponding file stored in themass storage 5.

The data server 4 which has received the above-described requestsearches for the requested file in the mass storage 5 and transfers thefile to the image obtaining means 61. The image obtaining means 61obtains the three-dimensional moving image V1 and the three-dimensionalultrasonic moving image V2 contained in the file transferred from thedata server 4 and stores them in the memory (S01).

Subsequently, the characteristic part extracting means 62 extracts, asthe predetermined characteristic part, the mitral valve MV, which is aheart valve located between the left ventricle LV and the left atriumLD, from each of the three-dimensional moving image V1 and thethree-dimensional ultrasonic moving image V2 (S02).

In this example, the method taught in R. I. Ionasec et al.,“Patient-Specific Modeling and Quantification of the Aortic and MitralValves From 4-D Cardiac CT and TEE”, IEEE TRANSACTIONS ON MEDICALIMAGING, VOL. 29, NO. 9, pp. 1636-1651, 2010, is applied to theoperations to extract the characteristic part and obtain the phases fromthe three-dimensional moving image V1 and the three-dimensionalultrasonic moving image V2.

The characteristic part extracting means 62 segments the mitral valve MVof the heart captured in the moving images V1 and V2 in time series forat least one period of heart beat according to the method taught in theabove-mentioned Non-Patent Document, and extracts information foridentifying the position of each sample point on the contour of themitral valve MV in each frame image forming the moving images V1 and V2.

Then, the phase obtaining means 63 obtains the phase of heart beat ofeach frame image based on the position of each sample point on thecontour of the mitral valve MV captured in the moving images V1 and V2(S03).

The one period of heart beat include a systole and a diastole. At theend of the systole, the aortic valve AV changes from the open state tothe closed state and the mitral valve MV starts to open from the closedstate. At the end of the diastole, the mitral valve MV changes to theclosed state and the aortic valve starts to open from the closed state.Using this nature, the phase obtaining means 63 identifies the end ofdiastole and the end of systole to identify the phases of heart beat.

In this embodiment, the shape of the mitral valve MV captured in each ofthe moving images V1 and V2 is obtained at predetermined time intervalsusing predetermined parameters according to the method taught in theabove-mentioned Non-Patent Document, and the state of opening andclosing of the mitral valve MV is identified based on the shape of themitral valve MV to obtain the phase of heart beat corresponding to thestate of opening and closing of the mitral valve MV. In this embodiment,for each of the moving images V1 and V2, the state of opening andclosing of the mitral valve MV and the predetermined parametersrepresenting the shape of the mitral valve MV are associated with eachother and stored. The phase obtaining means 63 identifies, for eachmoving image V1, V2, a frame in which the mitral valve MV has changedfrom the open state to the closed state as a frame corresponding to theend of diastole of the heart beat. Also, the phase obtaining means 63identifies, for each moving image V1, V2, a frame in which the mitralvalve MV has changed from the closed state to the open state(start-to-open state) as a frame corresponding to the end of systole ofthe heart beat. The predetermined parameters representing the shape ofthe mitral valve MV may, for example, be distances between specificsample points on the contour of the mitral valve MV.

Then, the associating means 64 temporally associates the frames formingthe moving images V1 and V2 with each other such that the moving imagesV1 and V2 are aligned with each other with respect to the phasescorresponding to the end of systole and the end of diastole (theassociating means 64 may perform interpolation in the time axisdirection, as necessary) (S04). In this example, the frame images of theimages V1 and V2 showing the same phase are associated with each other,and the spatial positions of the same characteristic part shown in theassociated frame images are associated with each other.

In the above-described operation, in the case where the number of framesof the three-dimensional moving image V1 and the number of frames of thethree-dimensional ultrasonic moving image V2 for one period differ fromeach other, the associating means 64 associates the frames of theseimages using the moving image having the smaller number of frames forone period as the reference. For example, the frames of the moving imagehaving the greater number of frames for one period may be appropriatelydecimated, as necessary. Further, in the case where the phases of theassociated frames are slightly out of alignment, interpolation may beperformed using a known method so that each pair of corresponding frameimages of the moving images shows the same phase. For example, the phaseof each frame image forming one of the moving images may be obtained,and then, using frame images of the other of the moving images beforeand after the obtained phase, an interpolated frame image of the otherof the moving images having the shape corresponding to the obtainedphase may be generated by a known method, to associate the frame imagesof the one of the moving images with the thus generated frame images ofthe other of the moving images such that each pair of associated frameimages shows the same phase.

The image generating means 65 generates volume rendered images for aseries of frame images extracted from the three-dimensional moving imageV1 by the above-described operation. For a series of frame imagesextracted from the three-dimensional ultrasonic moving image V2, theimage generating means 65 generates images by transforming thecoordinate system of the three-dimensional ultrasonic moving image V2into the coordinate system of the three-dimensional moving image V1 sothat the images V1 and V2 show the characteristic part associated by theassociating means 64 in the same position, the same direction and thesame size. Then, the image generating means 65 generates thesuperimposed image of the moving images V1 and V2 by a known method andstores the superimposed image in the storage 5 (S05).

Specifically, the image generating means 65 achieves the spatialalignment by transforming the coordinate system of one of the imagesinto the coordinate system of the other of the images so that the imagesshow the same characteristic part in the same spatial position based onthe position associated by the associating means 64, and appropriatelycorrecting the transformed coordinate system so that the images show thesame characteristic part in the same spatial position, the samedirection and the same size. It should be noted that, during the spatialposition alignment, the associating means 64 obtains pixel spacinginformation of the three-dimensional moving image V1 and thethree-dimensional ultrasonic moving image V2 from the DICOM headerinformation of each image, and enlarges or reduces the moving images V1and V2, as appropriate, based on the pixel spacing information toprovide the series of frame images extracted from the three-dimensionalmoving image V1 and the three-dimensional ultrasonic moving image V2with the same pixel spacing.

Further, as shown in FIG. 3, the superimposed image generated by theimage generating means 65 of this embodiment shows voxel values based onthe three-dimensional moving image V1 at a predetermined transparency byvolume rendering, and as shown by arrow C in FIG. 3, shows voxel valuesand the direction of blood flow based on the three-dimensionalultrasonic moving image V2 by the known color Doppler method.

As the method for generating the superimposed images of the series offrame images extracted from the three-dimensional moving image V1 andthe three-dimensional ultrasonic moving image V2, the image generatingmeans 65 may apply any of various known generation method that allowsdisplay of the superimposed images of the series of frame imagesextracted from the moving images V1 and V2 such that the superimposedimages show the same characteristic part in the same spatial position,the same direction and the same size.

The display controlling means 66 obtains the superimposed moving image(superimposed frame images) generated by the image generating means 65,and causes the display 7 to display the superimposed image, as shown inFIG. 3 (S06).

As described above, according to this embodiment, the phases of theperiodic motion of the body part that makes a predetermined motion areobtained based on the shape of the characteristic part captured in thethree-dimensional moving image V1 and the ultrasonic moving image V2 andthe three-dimensional moving image V1 and the ultrasonic moving image V2are aligned with each other with respect to the phases to achieve thespatial alignment based on the position of the characteristic partcaptured in the moving images. Therefore, even in a case where theelectrocardiographic data of one of the moving images is not available,the moving images can appropriately be associated with each other.Further, by generating and displaying the superimposed image of theassociated moving images, the user can understand the object ofobservation by compensating for low resolution areas of the ultrasonicmoving image V2 with the high spatial resolution of thethree-dimensional moving image V1, and can easily understand theinformation that is obtained only from the ultrasonic moving image atthe same time. Therefore, the user can efficiently and accuratelyconduct the imaging diagnosis.

In this embodiment, the superimposed image of the ultrasonic movingimage V2, which is shown by the color Doppler method based on theDoppler shift of blood flow, and the three-dimensional moving image V1is displayed. Therefore, the user can preferably understand the bodypart of interest at high spatial resolution based on thethree-dimensional moving image and the blood flow information, which isobtained only from the ultrasonic moving image, at the same time in anintuitive manner.

In this embodiment, the shape of the characteristic part isautomatically recognized to be extracted from the three-dimensionalmoving image and the three-dimensional ultrasonic moving image toeliminate the need of manual operation by the user to extract thecharacteristic part, and thus the shape of the characteristic part canbe extracted efficiently and easily.

In the case where the body part that makes a periodic motion is theheart and the predetermined characteristic part is any of the valves ofthe heart, the periodic motion of the heart is accurately identifiedbased on the state of opening and closing of the valves of the heart,thereby preferably obtaining the phases. Further, in this case, thephase obtaining means obtains the phases by identifying the end ofdiastole and/or systole of the heart based on the state of opening andclosing of the mitral valve and/or the aortic valve among the valves ofthe heart. Therefore, more accurate identification of the periodicmotion of the heart is achieved based on the change of the shape of thecharacteristic part in response to the heart beat. Still further, inthis embodiment, the associating means 64 aligns the moving images V1and V2 with respect to the phases of heart beat (the end of systole andthe end of diastole), thereby more accurately associating the movingimages V1 and V2 with each other.

In this embodiment, the phases of both the three-dimensional movingimage V1 and the ultrasonic moving image V2 are obtained by automaticrecognition, and therefore the phases are easily and accurately obtainedand associated. Alternatively, for one of the three-dimensional movingimage and the three-dimensional ultrasonic moving image, the phases ofthe periodic motion may be identified based on the shape of thecharacteristic part, and for the other of the moving images, the phaseof the periodic motion may be obtained based on the DICOM headerinformation, or the like. In this case, the automatic recognition may beapplied to only one of the moving images to minimize increase ofcomputational load and efficiently obtain the phases of the movingimages.

In this embodiment, the image generating means 65 generates thesuperimposed image by obtaining the pixel spacing from the accompanyinginformation of each of the three-dimensional moving image V1 and theultrasonic moving image V2, and providing the moving images with thesame pixel spacing based on the obtained pixel spacing. This facilitatesobtaining the pixel spacing of each moving image and accuratelyproviding the moving images of the same size to generate thesuperimposed image.

In this embodiment, the above-described image processing is carried outbased on the three-dimensional moving image V1 taken with a CT or MRapparatus and the three-dimensional ultrasonic moving image V2, and thisprovides the user with more detailed understanding of the object ofobservation.

The characteristic part extracting operation according to thisembodiment may be achieved by applying the method taught in Y. Zheng etal., “Four-Chamber Heart Modeling and Automatic Segmentation for 3DCardiac CT Volumes Using Marginal Space Learning and SteerableFeatures”, IEEE TRANSACTIONS ON MEDICAL IMAGING, Vol. 27, pp. 1668-1681,2008. It should be noted that the characteristic part extracting means62 may apply any of known various methods that can extract acharacteristic part of a structure from the two three-dimensional movingimages V1 and V2. For example, the user may manually input the positionand shape of the characteristic part, such as the valves of the heart,using a mouse, or the like, for each of the three-dimensional movingimages V1 and V2, and the image processing device may obtain such inputsto extract the position and the shape of the characteristic part.

The phase obtaining means 63 may determine the phases of heart beat byusing any method that uses the nature that the aortic valve AV changesfrom the open state to the closed state and the mitral valve MV startsto open from the closed state at the end of systole, and the mitralvalve MV changes to the closed state and the aortic valve starts to openfrom the closed state at the end of diastole. For example, a period froma point when the mitral valve MV starts to open from the closed state(the end of systole) to a point when the mitral valve MV again starts toopen from the closed state (the end of systole) may be detected as theone period of heart beat to associate the images V1 and V2 such that theimages are aligned with respect to the phase of the end of systole, or aperiod from a point when the mitral valve MV changes from the open stateto the closed state (the end of diastole) to a point when the mitralvalve MV again changes from the open state to the closed state (the endof diastole) may be detected as the one period of heart beat toassociate the images V1 and V2 such that the images are aligned withrespect to the phase of the end of diastole. Alternatively, for example,the phases of heart beat may be determined based on the state of openingand closing of the aortic valve AV, in place of the mitral valve MV, orinformation of the state of opening and closing of the mitral valve MVand information of the state of opening and closing of the aortic valveAV may be weighted to be used to determine the phase of heart beat.

In the case where any of various characteristic parts, such as the leftventricle LV, the left atrium LA, the right ventricle RV, the rightatrium RA, the valves MV, AV, PV and TV and the apex AC of the heart, asshown in FIG. 3, or any combination of these characteristic parts isused as the predetermined characteristic part to identify the periodicmotion of the heart based on the periodical change of the shapedepending on the phase of heart beat, similarly to this embodiment, thephases are accurately obtained. In the case where more than onecharacteristic parts are used to identify the periodical motion of theheart, the phases are more accurately obtained based on the more thanone pieces of information.

If the moving image V1 and/or the moving image V2 contains two or moreperiods of periodic motion, the phase obtaining means 63 may arbitrarilyspecify a period used to associate the moving images V1 and V2 with eachother. The phase obtaining means 63 according to this embodimentreceives an input by the user via a mouse and/or keyboard to identifyone of the periods specified by the user by using any known method. Forexample, period selection buttons corresponding to the two or moreperiods contained in the three-dimensional moving image V1 or thethree-dimensional ultrasonic moving image V2 may be displayed to receivethe selection of period by the user, or the user may be prompted toinput the start time of one of the periods contained in thethree-dimensional moving image V1 or the three-dimensional ultrasonicmoving image V2 via a keyboard, or the like, and the phase obtainingmeans 63 may receive the selection of period by the user.

FIG. 4 shows an example of the displayed superimposed image according toa modification of the above-described embodiment. Although theabove-described embodiment is described in conjunction with thethree-dimensional ultrasonic moving image V2 as an example, it isapparent for those skilled in the art that the invention is similarlyapplicable to a two-dimensional moving image as long as the image showsa cross section showing a recognizable characteristic part included in abody part, such as a cross section P showing the ventricles LV and RV,the atriums LA and RA, the valves MV, AV, PV and TV and the apex AC ofthe heart, as shown in FIG. 4. The user can observe, with respect to apredetermined cross section including the characteristic part, a highspatial resolution image taken with a CT or MR apparatus, and canunderstand information, such as information of blood flow, which isobtained only from a two-dimensional moving image taken with anultrasonic diagnostics apparatus at the same time. This facilitates theuser to accurately conduct the imaging diagnosis.

It should be noted that the present invention is not limited to thisembodiment. For example, the predetermined body part may be any bodypart that makes a predetermined periodical motion, such as flexion andextension of a knee joint. In the case where the invention is applied toflexion and extension of the knee, or the like, one or more partsforming the knee joint may be segmented to obtain parametersrepresenting the state of flexion and extension of the knee, such asdistances between predetermined points on the thigh bone and theshinbone, from the segmented parts, and the phases of the periodicalmotion from the flexed state to the extended state may be obtained basedon the parameters representing the state of flexion and extension.

It should be noted that the alignment of the three-dimensional movingimage V1 taken with a CT or MR apparatus and the ultrasonic moving imageV2 may be achieved by transforming the coordinate system of thethree-dimensional moving image V1 taken with a CT or MR apparatus intothe coordinate system of the ultrasonic moving image V2.

The associating means 64 may associate the phases of thethree-dimensional moving image V1 taken with a CT or MR apparatus andthe ultrasonic moving image V2 for only a part of one period of periodicmotion, for one period of periodic motion, or for two or more periods ofperiodic motion.

Although the embodiments of the present invention have been describedwith respect to the case where the image processing program of theinvention is implemented on a single diagnosis WS to cause the WS tofunction as the image processing device, the image processing programmay be installed on two or more computers in a distributed manner tocause the two or more computers to function as the image processingdevice.

1. An image processing device comprising: an image obtaining unit forobtaining a three-dimensional moving image showing a body part of apatient and an ultrasonic moving image showing the body part, the bodypart making a predetermined periodic motion; a characteristic partextracting unit for extracting, from a plurality of frame images formingthe obtained three-dimensional moving image and ultrasonic moving image,a predetermined characteristic part having a shape that changes inresponse to the periodic motion; a phase obtaining unit for obtainingphases of the periodic motion captured in the three-dimensional movingimage and the ultrasonic moving image, wherein, for at least one of thethree-dimensional moving image and the ultrasonic moving image, thephases are obtained based on the shape of the extracted characteristicpart; an associating unit for associating, for each phase, a position ofthe characteristic part in a frame image forming the three-dimensionalmoving image corresponding to the phase with a position of thecharacteristic part in a frame image forming the ultrasonic moving imagecorresponding to the phase based on the extracted characteristic partand the obtained phases; an image generating unit for generating asuperimposed image of the three-dimensional moving image and theultrasonic moving image by aligning, for each phase, a position of thecharacteristic part in a frame image forming the three-dimensionalmoving image corresponding to the phase with a position of thecharacteristic part in a frame image forming the ultrasonic moving imagecorresponding to the phase based on the associated positions of thecharacteristic part and the phases; and a display controlling unit fordisplaying the generated superimposed image on a display device.
 2. Theimage processing device as claimed in claim 1, wherein thecharacteristic part extracting unit automatically extracts thecharacteristic part.
 3. The image processing device as claimed in claim1, wherein the body part making the periodic motion is a heart, and thecharacteristic part is any of ventricles, atriums, muscles, valves andan apex of the heart.
 4. The image processing device as claimed in claim3, wherein the phase obtaining unit obtains the phases based on a stateof opening and closing of any of the valves of the heart.
 5. The imageprocessing device as claimed in claim 4, wherein the phase obtainingunit obtains the phases by identifying an end of diastole and/or systoleof the heart based on the state of opening and closing of a mitral valveand/or an aortic valve among the valves of the heart.
 6. The imageprocessing device as claimed in claim 1, wherein, for at least one ofthe three-dimensional moving image and the ultrasonic moving image, thephase obtaining unit obtains the phases from accompanying information ofthe moving image.
 7. The image processing device as claimed in claim 1,wherein the image generating unit generates the superimposed image byobtaining pixel spacing from accompanying information of each of thethree-dimensional moving image and the ultrasonic moving image andproviding the moving images with the same pixel spacing based on theobtained pixel spacing.
 8. The image processing device as claimed inclaim 1, wherein the image generating unit generates the superimposedimage by superimposing, on the three-dimensional moving image, theultrasonic moving image shown by a color Doppler method based on Dopplershift of blood flow.
 9. The image processing device as claimed in claim1, wherein the ultrasonic moving image is a moving image showing a crosssection including the characteristic part.
 10. An image processingmethod comprising: obtaining a three-dimensional moving image showing abody part of a patient and an ultrasonic moving image showing the bodypart, the body part making a predetermined periodic motion; extracting,from a plurality of frame images forming the obtained three-dimensionalmoving image and ultrasonic moving image, a predetermined characteristicpart having a shape that changes in response to the periodic motion;obtaining phases of the periodic motion captured in thethree-dimensional moving image and the ultrasonic moving image, wherein,for at least one of the three-dimensional moving image and theultrasonic moving image, the phases are obtained based on the shape ofthe extracted characteristic part; associating, for each phase, aposition of the characteristic part in a frame image forming thethree-dimensional moving image corresponding to the phase with aposition of the characteristic part in a frame image forming theultrasonic moving image corresponding to the phase based on theextracted characteristic part and the obtained phases; generating asuperimposed image of the three-dimensional moving image and theultrasonic moving image by aligning, for each phase, a position of thecharacteristic part in a frame image forming the three-dimensionalmoving image corresponding to the phase with a position of thecharacteristic part in a frame image forming the ultrasonic moving imagecorresponding to the phase based on the associated positions of thecharacteristic part and the phases; and displaying the generatedsuperimposed image on a display device.
 11. A non-transitory storagemedium containing an image processing program for causing a computer tofunction as: an image obtaining unit for obtaining a three-dimensionalmoving image showing a body part of a patient and an ultrasonic movingimage showing the body part, the body part making a predeterminedperiodic motion; a characteristic part extracting unit for extracting,from a plurality of frame images forming the obtained three-dimensionalmoving image and ultrasonic moving image, a predetermined characteristicpart having a shape that changes in response to the periodic motion; aphase obtaining unit for obtaining phases of the periodic motioncaptured in the three-dimensional moving image and the ultrasonic movingimage, wherein, for at least one of the three-dimensional moving imageand the ultrasonic moving image, the phases are obtained based on theshape of the extracted characteristic part; an associating unit forassociating, for each phase, a position of the characteristic part in aframe image forming the three-dimensional moving image corresponding tothe phase with a position of the characteristic part in a frame imageforming the ultrasonic moving image corresponding to the phase based onthe extracted characteristic part and the obtained phases; an imagegenerating unit for generating a superimposed image of thethree-dimensional moving image and the ultrasonic moving image byaligning, for each phase, a position of the characteristic part in aframe image forming the three-dimensional moving image corresponding tothe phase with a position of the characteristic part in a frame imageforming the ultrasonic moving image corresponding to the phase based onthe associated positions of the characteristic part and the phases; anda display controlling unit for displaying the generated superimposedimage on a display device.